Who searched My Blog? Keyword Research Tutorials¶
Objectives¶
- I have a blog about life in the US as a Korean (http://selim.tistory.com).
- The topics of the postings are diverse about life information, traveling US, cooking, grad-life, self-care, and Urbana-Champaign(University of Illinois).
- I gathered user data for around two months (2021.9.1 - 2021.11.15) through Google analytics.
- User information and keywords were analyzed to understand and find users' interests.
Methods¶
- User information (i.e., country, region) and searched keywords (from 2021.9.1 - 2021.11.15) were extracted from Google analytics tools.
- For descriptive analysis,
- Using Tableau, I plotted user information by pageviews and retention time.
- For keyword research,
- Using Text Mining methods, general topics were analyzed by the most frequently searched keywords (separately in Korean and English)
Summary of Findings¶
- People viewed my blog nearly all over the world.
- Top 2 countries were South Korea and U.S.A.
- Because most postings were written in Korean, most keywords were searched in Korean.
- The keywords searched most were similar between users in South Korea and U.S.A.
- Top 3 topics of User's Interests:
- 1st topic was local information (such as, 'USA', 'Chicago', 'San Jose', 'LA', or 'Jamsil'(a district in Seoul)).
- 2nd topic was product reviews ('Recommdation', 'Review', 'Instant Pot', 'US grocery market', 'Trader Joe's', 'Shein').
- 3nd topic was diet/health ('Keto', 'LCHF', 'Review', 'diet', 'stevia').
In [2]:
from IPython.display import Image
Image("clouds.png")
Out[2]:
Descriptive Analysis for Understanding Users (enabled by Tableau)¶
- Using Tableau, I plotted user's information.
Geographical Information of Users¶
- Users viewed my blog all over the world from all continents.
- Top 2 regions were Seoul and California.
- Top 2 countries were South Korea and U.S.A.
Time, Country and Page Views¶
- The time users viewed my blog reflects the time zones they are in.
Source, Country, and Page Views¶
- "Google Search" is the most frequent and common method that users were getting into my blog, regardless of country.
- Korean users used "Naver" for the second most, as "Naver" is the most popular search engine in South Korea.
- Daum (another Korean search engine) and Youtube are followed.
Searched Keywords by Pageviews and Average Time Retention¶
- The most frequent word is "미국", which means "USA" in Korean.
- However, the list of searched keywords were dispersed as various topics.
- Therefore, I investigated top topics searched as keywords through the text mining methods from the keywords lists saved as .csv files.
Keywords Research (Text Mining)¶
The keywords that were searched most frequently were examined using text mining methods (konlpy for Korean keywords and ntlk/word cloud for English keywords).
Using wordcloud, the most frequent keywords were visualized separately in Korean and English.
1. Finding topics from Korean keywords¶
In [17]:
# Import required libraries
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
In [132]:
df = pd.read_csv('keywords.csv')
In [19]:
df.head()
Out[19]:
In [20]:
df.shape
Out[20]:
In [21]:
import re
def apply_regular_expression(text):
hangul = re.compile('[^ ㄱ-ㅣ 가-힣]')
result = hangul.sub('', text)
return result
In [22]:
apply_regular_expression(df['Keyword'][0])
Out[22]:
In [24]:
from konlpy.tag import Okt
from collections import Counter
okt = Okt()
nouns = okt.nouns(apply_regular_expression(df['Keyword'][0]))
nouns
Out[24]:
In [ ]:
# Create corpus
corpus = "".join(df['Keyword'].tolist())
In [ ]:
# Apply regular expressions
apply_regular_expression(corpus)
# Extract Tokens form Corpus
nouns = okt.nouns(apply_regular_expression(corpus))
In [28]:
counter = Counter(nouns)
In [29]:
counter.most_common(10)
Out[29]:
In [30]:
#Delete Stopwords
available_counter = Counter({x: counter[x] for x in counter if len(x) > 1})
available_counter.most_common(10)
Out[30]:
Most Common Keywords in Korean (N of Times Searched)¶
- 'USA' (391)
- 'Instant' (113)
- 'Chicago' (72)
- 'Recommand' (72)
- 'Korea' (71)
- 'Jamsil' (a district of Seoul) (61)
- 'San Jose' (58)
- Market (57)
- Women (53)
- Shoulder (53)
In [31]:
#Delete Stopwords from an external dictionary
stopwords = pd.read_csv("https://raw.githubusercontent.com/yoonkt200/FastCampusDataset/master/korean_stopwords.txt").values.tolist()
In [42]:
!pip install wordcloud
In [33]:
import nltk
nltk.download('stopwords')
nltk.download('punkt')
Out[33]:
In [43]:
from wordcloud import WordCloud, STOPWORDS
import numpy as np
import nltk
from PIL import Image
In [34]:
text = open('keywords.csv').read()
circle = np.array(Image.open('circle_1.jpeg'))
stopwords = set(STOPWORDS)
stopwords.add("said")
In [35]:
print('The number of stopwords:',len(nltk.corpus.stopwords.words('english')))
print(nltk.corpus.stopwords.words('english')[:40])
In [36]:
plt.figure(figsize=(8,8))
plt.imshow(circle, cmap=plt.cm.gray, interpolation='bilinear')
plt.axis('off')
plt.show()
In [46]:
# Install Korean fonts
!apt-get update -qq
!apt-get install fonts-nanum* -qq
In [47]:
text = open('keywords.csv').read()
circle = np.array(Image.open('circle_1.jpeg'))
stopwords = set(STOPWORDS)
stopwords.add("said")
fontpath = '/usr/share/fonts/truetype/nanum/NanumBarunGothic.ttf'
wc = WordCloud(font_path = fontpath, background_color='white', max_words=2000, mask=circle,
stopwords = stopwords)
wc = wc.generate(text)
wc.words_
Out[47]:
In [48]:
plt.figure(figsize=(12,12))
plt.imshow(wc, interpolation='bilinear')
plt.axis('off')
plt.show()
2. Finding English Keyword Topics¶
In [133]:
# Finding English Keywords
# Defining Data
list_eng = []
#Delete Special cases
def cleanText(readData):
text = re.sub('[-=+,#/\?:^$.@*\"※~&%ㆍ!』\\‘|\(\)\[\]\<\>`\'…》]', ' ', readData)
return text
for i in df['Keyword']:
# Deleting Korean words
text = re.sub('[^a-zA-Z]',' ',i).strip()
#Delete Special cases
text = cleanText(text)
#Capitalize all words
text = text.upper()
# Deleting empty rows
if(text != ''):
list_eng.append(text)
print(list_eng)
In [119]:
# dictionary of lists
dict = {'name': list_eng}
df = pd.DataFrame(dict)
# saving the dataframe
df.to_csv('eng_keywords.csv')
In [121]:
corpus = pd.DataFrame(list_eng)
text = " ".join(corpus[0].tolist())
text
Out[121]:
In [77]:
from nltk.tokenize import word_tokenize
tokenized_word=word_tokenize(text)
print(tokenized_word)
In [78]:
from nltk.probability import FreqDist
fdist = FreqDist(tokenized_word)
print(fdist)
In [80]:
fdist.most_common(10)
Out[80]:
In [81]:
# Frequency Distribution Plot
import matplotlib.pyplot as plt
fdist.plot(30,cumulative=False)
plt.show()
In [ ]:
#removing stopwords
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
stop_words = set(stopwords.words('english'))
word_tokens = word_tokenize(text)
result = []
for token in word_tokens:
if token not in stop_words:
result.append(token)
text = result
In [83]:
counter = Counter(text)
counter.most_common(10)
Out[83]:
In [84]:
#Delete Stopwords
available_counter = Counter({x: counter[x] for x in counter if len(x) > 1})
available_counter.most_common(10)
Out[84]:
In [125]:
text_eng = open('eng_keywords.csv').read()
circle = np.array(Image.open('circle_1.jpeg'))
stopwords = set(STOPWORDS)
wc_e = WordCloud(collocations=False, background_color='white', max_words=2000, mask=circle, stopwords=stopwords)
wc_e = wc_e.generate(text_eng)
wc_e.words_
Out[125]:
In [126]:
plt.figure(figsize=(10,10))
plt.imshow(wc_e, interpolation='bilinear')
plt.axis('off')
plt.show()
As you can see in the word clous, I can see the top 3 topics of User's Interests:¶
- 1st topic was local information (such as, 'USA', 'Chicago', 'San Jose', 'LA', or 'Jamsil').
- 2nd topic was consumer's reviews ('Recommdation', 'Review', 'Instant Pot', 'US grocery market', 'Trader Joe's', 'Shein').
- 3nd topic was diet/health ('Keto', 'LCHF', 'Review', 'diet', 'stevia').
In [ ]: